Video conferencing during real time collaboration on a virtual canvas

ABSTRACT

A method of collaborating between a first computer associated with a first display at a first location and a second computer associated with a second display at a second location may include establishing a connection between the first and second computers, opening a virtual canvas on the first computer, the virtual canvas to be displayed on the first and second displays simultaneously, and sending an object between the first and second computers by sending data associated with the object on the virtual canvas stored on the first computer to the second computer to be stored locally, thereby creating a shared canvas on which objects are at a single location. Video conferencing may include sending a first live video stream from the first display to the second display to be viewed in a first video conference window separate from the virtual canvas on the second display and vice versa.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of PCT filingPCT/US2019/035620, filed Jun. 5, 2019, which claims priority to U.S.Provisional Application No. 62/680,903 filed Jun. 5, 2018, and entitled:“Video Conferencing During Real Time Collaboration on a Virtual Canvas,”each of which are incorporated herein by reference in their entirety.

DESCRIPTION OF RELATED ART

There are currently different solutions to collaborating over multiplelocations. Web-based conferencing mirrors an entire screen of a firstcomputer to the screen of other computers. The web-based conferencingsoftware allows any of the computers connected to be the “presenter.”All of the other users connected can view the presenter's screen ontheir computer. However, there is the limitation that there is only onepresenter who is mirroring their display at a time. The computer of thepresenter sends the whole screen to the other computers at a refreshrate, i.e., streams the entire screen to a screen at another location.

Cloud based collaboration tools host a session on a cloud server andprovide a link that client computers use to log in to the session on thecloud server. Multiple documents can be uploaded into the canvas ofsession on the cloud server and then viewed and/or downloaded by anyclient computers connected to the session. Each location in a sessioncan then use an application to edit a document or file in the sessiondirectly on the cloud server. While more than one location may edit thecontent in the session at a time, when something is being changed on ascreen at one location, the computer at that location waits until theaction is finished and then sends the information to the cloud server. Ahost session can open a canvas that is much larger than an individualscreen. Each user at each location that joins the session can editdocuments on the canvas.

For example, a user can perform a move or resize gesture on a window.After the gesture is complete, then the new size or location of thewindow is sent to the cloud server and the window is updated and sent toall locations. If an annotation, e.g., a line or circle, is drawn, oncecompleted, i.e., a finger or stylus is lifted from the display, thenthis info is sent to the cloud server and the drawn image is stored onthe canvas on the cloud server. For example, if a user at one locationhas a powerpoint file that they would like to share, they first uploadthe powerpoint file to the cloud. Then all of the slides in thepowerpoint file are converted to a useable file format, stored in thecloud, and are then available for viewing from other locations. Whileeverything is accessible in the cloud at any time in different timezones, e.g., asynchronous collaboration, when real time collaboration isdesired, this may not be sufficient. Further, while these solutions mayfunction well for many file types, live video feeds are not handled bythis solution.

SUMMARY

One or more embodiments are directed to a method of collaboratingbetween a first computer associated with a first display at a firstlocation and a second computer associated with a second display at asecond location. The method may include establishing a connectionbetween the first and second computers, opening a virtual canvas on thefirst computer, the virtual canvas to be displayed on the first andsecond displays simultaneously, and sending an object between the firstand second computers by sending data associated with the object on thevirtual canvas stored on the first computer from the first computer tothe second computer to be stored locally, thereby creating a sharedcanvas, wherein objects are at a single location on the shared canvas,sending a first live video stream from the first display to the seconddisplay to be viewed in a first video conference window separate fromthe virtual canvas on the second display, and sending a second livevideo stream from the second display to the first display to be viewedin a second video conference window separate from the virtual canvas onthe first display.

The method may further include connecting a mobile display device to oneof the first and second computers, sending a rendering of the sharedcanvas, and directly sending the first and second live video streams ina video conference window on the mobile display device separate from therendering.

Each of the first and second video conference windows is in a videoconferencing panel.

The method may further include selecting a video conference window andaltering the selected video conference window.

Altering the selected video conference window may include moving theselected video conference window to be adjacent a camera on a displayviewing the window.

Altering the selected video conference window may include enlarging thewindow.

Selecting the video conference window may include detecting an audiosignal associated with the video conference window.

The video conferencing panel may be stationary and the at least twovideo conference windows may be manipulatable.

The first and second video conference windows may be stationary, and theshared canvas and objects thereon may be manipulatable.

Each of the first and second video conference windows may be adjacent acamera associated with a display on which respective video conferencewindows are viewed.

The video conference window may be in a periphery of a display on whichrespective video conference windows are viewed.

The connection for the virtual canvas may provide information needed tosend the first and second live video streams.

The connection for the virtual canvas may provide information to send tothe first and second live video streams.

The virtual canvas may be stored in the cloud and the first and secondlive video streams may be transmitted in real-time.

The virtual canvas may be a restricted virtual canvas or an unrestrictedvirtual canvas.

The first live video stream may be displayed in the second videoconference window separate from the virtual canvas on the first display,and the second live video stream may be displayed in the first videoconference window separate from the virtual canvas on the seconddisplay.

One or more embodiment may be directed to a method of collaboratingbetween first and second mobile devices having sharing applicationdownloaded thereon and a host computer, the second mobile device beingremote from the first mobile device and the host computer. The methodmay include establishing a peer-to-peer connection between the first andsecond mobile devices and the host computer, opening a virtual canvas onthe host computer having a host display, the virtual canvas to be sharedby the first and second mobile devices and the host display, sending afirst object from the first mobile device to the virtual canvas bysending data associated with the first object to the host computer andsending the first object to the second mobile device by sending dataassociated with the first object from the host computer to the secondmobile device, sending a first live video stream from the first mobiledevice to the host device to be viewed in a first video conferencewindow separate from the virtual canvas on the host display, and sendinga second live video stream from the second mobile device to the firstmobile device to be the host device to be viewed in a second videoconference window separate from the virtual canvas on the host display.

The virtual canvas may be stored in the cloud.

Collaboration may include a second host computer having a second hostdisplay located near the second mobile device, the method furtherincluding establishing a peer-to-peer connection between the hostcomputer and the second host computer, wherein the virtual canvas to bedisplayed on the first and second displays simultaneously, anddisplaying the first and second live video streams on the second hostdisplay in first and second windows separate from the virtual canvas.

Sending the first and second live video streams may include using useraccounts associated with the first and second mobile devices.

Establishing the peer-to-peer connection may include generating a key bythe host device, reading the key by the first mobile device, sending thekey to the second mobile device, and entering the key into the secondmobile device.

One or more embodiments is directed to a method of collaborating betweena first computer associated with a first display at a first location anda second computer associated with a second display at a second location.The method may included establishing a connection between the first andsecond computers, opening a virtual canvas on the first computer, thevirtual canvas to be displayed on the first and second displayssimultaneously, and sending an object between the first and secondcomputers by sending data associated with the object on the virtualcanvas stored on the first computer from the first computer to thesecond computer to be stored locally, thereby creating a shared canvas,wherein objects are at a single location on the shared canvas, sending afirst live video stream from the first display to the second display tobe viewed in a first video conference window on the second display, andsending a second live video stream from the second display to the firstdisplay to be viewed in a second video conference window on the firstdisplay, wherein at least one of the first and second video conferencewindows is not on the virtual canvas.

One or more embodiments is directed to a collaboration system, includinga group display and a group display computer, the group display computerto provide a single display window at full screen or multiple displaywindows, the multiple display windows being in a fixed layout on thegroup display, wherein the group display computer is to receive mobiledevice data streams from at least two mobile devices runningcollaboration software over a wireless access protocol and to outputdisplay data streamed from both of the at least two mobile devices inseparate display windows of the multiple display windows, the groupdisplay computer to further indicate the presence of additional displaywindows that are not displayed on the group display, all display windowsbeing on a virtual canvas.

Annotations performed on the mobile device appear on the group displayand annotations performed on the group display appear on both of the atleast two mobile devices.

The fixed layout may include x rows and y columns, wherein x and y areboth equal to or greater than two, one of x and y is fixed, and anotherof x and y is not fixed.

Each window can contain a mobile device stream, a white board, a videoconference stream, and a snapshot of a stream.

The fixed layout may include four equally sized display windows.

The multiple display windows may be interchangeable.

When a snapshot of one of the at least two windows is taken, thesnapshot is displayed in another one of the multiple display windows.

Selecting an icon may switch between the single display window and themultiple display windows.

The group display and the group display computer may identify mobiledevices within connection range running collaboration software and todisplay an indication thereof on the group display, and, when theindication is activated, automatically connect to an associated mobiledevice to receive data streams therefrom over a wireless access protocoland to output display data streamed therefrom to the single displaywindow at full screen or in a display window of the multiple displaywindows.

The indication may include a meeting calendar listing mobile devicesassociated with all invitees.

A collaboration system may include a video conferencing panel, separatefrom the virtual canvas, on which video conference windows are displayedduring video conferencing.

One or more embodiments are directed to a collaboration system,including a group display and a group display computer, the groupdisplay computer to receive mobile device data streams from at least twomobile devices over a wireless access protocol and to output displaydata streamed from both of the at least two mobile in separate windowsof multiple display windows having a fixed configuration, the groupdisplay including a video conferencing panel including different videoconferencing applications to be selected, the video conferencing panelonly permitting video conferencing, screen sharing, and annotation.

The group display may display an icon that, when activated, opens thevideo conferencing panel that is otherwise hidden.

Selection of a video conferencing application in the video conferencingpanel starts a video conference.

The video conference window may be in one of the multiple displaywindows.

The video conferencing window may be in the video conferencing panelseparate from the multiple display windows.

BRIEF DESCRIPTION OF THE DRAWINGS

Features will become apparent to those of skill in the art by describingin detail exemplary embodiments with reference to the attached drawingsin which:

FIGS. 1 to 8 illustrate a display system in accordance with a firstembodiment;

FIGS. 9A to 9C illustrate differences between the first embodiment and asecond embodiment; and

FIGS. 10A to 10U illustrate a display system in accordance with thesecond embodiment.

DETAILED DESCRIPTION

Example embodiments will now be described more fully hereinafter withreference to the accompanying drawings; however, they may be embodied indifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey exemplary implementations to those skilled in the art.

Embodiments are directed to using video conferencing stream with acanvas such that the canvas may be manipulated without affecting thevideo conferencing stream. A Canvas is a virtual region that expands togreater than the physical area of a display region of a display, e.g.,any number of times the physical area up to infinite. The use of theCanvas allows additional files to be accessible and can be saved, butoff the display region. Further discussion of the canvas may be found inU.S. Pat. No. 9,596,319, which is hereby incorporated by reference inits entirety for all purposes. Gestures, such as pan, zoom and pinchgestures can be made to move and resize the scale of the Canvas,allowing the full canvas to be displayed at once or only a small sectionthereof.

In previous disclosures, a video conferencing stream occupied a windowon the canvas. Therefore, when the canvas was manipulated, the videoconferencing stream may be affected, e.g., may disappear altogether fromthe display region. However, users may want the video conferencingstream, e.g., a live video feed of a remote user, to remain on thedisplay region, e.g., at a same position on the display region, at alltimes. Therefore, according to embodiments set forth below, a live videofeed may be treated differently than other windows on the canvas.

First, FIG. 1 will provide a brief overview of the Canvas used at asingle location.

FIG. 1 illustrates a block diagram of a display system 100 interactingwith one or more mobile devices 200 a, 200 b, and so forth. The displaysystem 100 includes a Common Display 110, a Display Computer 120, and awireless router 130 serving as a wireless access point (WAP), allinterconnected. The Common Display 110 may be an LCD display, LEDdisplay, or other monitor that is capable of having an electronic videosignal as an input and converting the input to a visual image.

The Common Display 110 may include a display region 112 and a trayregion 114, e.g., below the display region. The Common Display 110 atouch sensor 116, e.g., overlaying an entirety of the Common Display110, that it is sensitive to touch inputs including taps and gestures.

Information regarding a Machine Identifier 122 of the Display Computer120 and the digital information to be displayed on the Common Display110 may be sent from the Display Computer 120 to the Common Display 110.This digital information can be within windows or Mobile Device Windows(MDWs), e.g., editable windows, or on the entire screen of displayregion 112 of the Common Display 110. In addition, there may be windowsdisplaying contents from mobile devices (MD) or other appropriate mobiledevice icons (MDI) 220 a, 220 b, e.g., a thumbnail of what is displayedon the mobile device, in the tray region 114 on the Common Display 110,e.g., at a lower region thereof. The tray region 114 may be a region onwhich the MDWs cannot be zoomed and pinched, annotated, and so forth,but may be dragged, tapped or tossed onto the display region 112, e.g.,to open an MDW corresponding to the MDI, and/or to receive MDWs from thedisplay region 112 to transmit that MDW to the mobile devicecorresponding to the MDI. The tray region 114 may also include anygraphical user interface (GUI) icons to be used on the Common Display110. Further, selecting a MDW on the canvas may bring up one or moreadditional GUIs.

To connect to the Display Computer 120, a user of a mobile device maydownload a display application 210 a, 210 b thereon to assist inconnecting the to the display computer 120. Such an application may bewritten for each common operating system and is herein referred to as anAirConnect™ App. Instructions for downloading the AirConnect™ App may bereadily viewable, e.g., on or adjacent the common display 110, or aregion to be scanned, e.g., a bar code, quick response (QR) code, and soforth, using a mobile device QR, so that once scanned, the AirConnect™App could be downloaded. Once the AirConnect™ App is downloaded, then auser can launch the AirConnect™ App and then enter the MachineIdentifier 122 associated common display 110. The Machine Identifier 122may by an IP address or other alphanumeric code associated with theDisplay Computer 120. The Machine Identifier 122 may be simply displayedon the Common Display 110, in which case the user of the AirConnect™ Appmay simply enter the Machine Identifier 122 when prompted by theAirConnect™ App on their Mobile Device. Alternatively, the MachineIdentifier 122 may be automatically transferred to the Mobile Deviceeither by displaying a QR code on the Common Display 110 or bytransmitting through bluetooth or wireless communication.

A virtual canvas (Canvas) that exists within the software on the DisplayComputer 120 may be much larger than the physical display region 112 onthe Common Display 110. Any section of the Canvas can be viewed on thedisplay region 112. Suppose that at first the entire Canvas is viewed onthe display region 112. Then by zooming into regions, a portion of theCanvas can be viewed. By pinching, zooming and panning with gestures onthe touch sensor 116, a user can zoom into various regions of the Canvasto be shown on the display region 112. The system can be configured sothat the MDW can be moved and manipulated. For example, if the CommonDisplay 110 includes the touch sensor 116, then touch gestures such aspan, zoom and pinch may be used to move and/or resize the MDWs.Snapshots of the MDWs can be taken from the Common Display, with the useof touch gestures. Annotations could be performed on the MDWs. Furtherdetails may be found, e.g., in U.S. patent application Ser. No.14/540,946.

Now, a brief overview of the Canvas as used in multiple locations. Asillustrated in FIG. 2, when operating at multiple locations, hereillustrated as two for simplicity, when the screen on the displays istreated as a Virtual Canvas, rather than an entire screen, as disclosed,for example, in U.S. Pat. No. 9,596,319, issued on Mar. 14, 2017, whichis hereby incorporated by reference for all purposes, individual objectsor windows on the Virtual Canvas on a display Display1 410 at a firstlocation Location 1 running on a first computer COMP1 420 may betransferred to a second computer COMP2 520 associated with a displayDisplay2 510 at a second location Location 2 to be displayed on aVirtual Canvas on the display Display2 510, and vice versa. Instead ofstoring the objects or windows in the cloud, a shared canvas may beestablished to be accessed by both computers COMP1 420 and COMP2 520running the displays using the Virtual Canvas software noted below andthe objects can be transferred directly, e.g., using a peer-to-peerconnection. This peer-to-peer connection may be established through arelay server, e.g., in the cloud, which does not store anything, butjust relays that data or the connection maybe direct peer-to-peerwithout going through a relay server.

As used herein, peer-to-peer connection means that the computers formingthe peer-to-peer connection communicate with each other, either directlyor through a relay server. When a relay server is used, the relay serverjust relays data in real time to facilitate communication, but does notrely on storing data to transfer information. In contrast, incloud-based communication protocol, each local computer communicatesinformation to a cloud-based server. The cloud-based server has themaster information or the master canvas and all objects on the mastercanvas. Periodically, each local computer requests from the cloud serverif any changes have been made to the master canvas and if so, asynchronization occurs between the local computer and the cloud server.

In accordance with embodiments, initially a virtual session is startedat a first location, with a computer, e.g., Comp1 420 or Comp2 520, thatis running Virtual Canvas software, e.g., ThinkHub® multisite softwareby TIV®. In particular, aVirtual Canvas that exists within the softwarethat may be much larger than the physical display region 412, 512 onDisplay1 410, 510. Any section of the Virtual Canvas can be viewed onthe display region 412, 512 of the Display1 410, 510. Suppose that atfirst the entire Virtual Canvasis viewed on the display region 412, 512and that each of the displays 410, 510 include a touch or gesture sensor416, 516 associated therewith. Then by zooming into regions, a portionof the Virtual Canvas can be viewed. By pinching, zooming and panningwith gestures detected by the touch sensor 416, 516, a user can zoominto various regions of the Virtual Canvas to be shown on the displayregion 412, 512.

The Virtual Canvas, is a virtual region that expands to greater than thephysical area of the display regions 412, 512, e.g., any number of timesthe physical area up to infinite. The use of the Virtual Canvas allowsadditional files to be accessible and can be saved, but off the displayregions 412, 512. Gestures, such as pan, zoom and pinch gestures can bemade to move and resize the scale of the Virtual Canvas, allowing thefull Virtual Canvas to be displayed at once or only a small sectionthereof.

After connection is established, then a user, e.g., at Comp1 420 startsa Multisite session and gives it a name, e.g., New1, as may be seen in awindow 418 on Display1 410. Then, a user at a second location with acomputer, e.g., Comp 2 520, that is also running the Virtual Canvas,software can see a list of active Multisite sessions that it haspermission to access, e.g., New1 as shown in window 518 on Display2 510.Then, the user at the second location taps on a button or on the name inthe list on the Virtual Canvas, to join the session. Once the secondcomputer joins the session, the connection is established between Comp1and Comp2, and the data for all the objects on Comp1 in the VirtualCanvas is then sent to Comp 2, and vice versa, such that the VirtualCanvas is now a shared canvas.

While the above has shown the displays displaying the objects at thesame locations on the two displays, this is true if the displays areboth displaying the same views of the Virtual Canvas. In other words,the objects are at the same location on the Virtual Canvas, differentviews of the Virtual Canvas, may be viewed on the different displays.For example, if Display 2 were viewing the same general region of theVirtual Canvas, but with a different zoom factor and slightly differentcenter point, then on Display 2 W1 and W2 would appear on the sameplaces on the Virtual Canvas, but different location on the Display andwith different scale factors. Further, one display may be viewing adifferent portion of the shared canvas without altering the view on theother display. Alternatively, changing a portions being viewed on onedisplay may alter that view on the other display.

As each object is stored locally, when an action is performed on thatobject, e.g., annotation, resizing, moving, and so forth, only that dataregarding the change is sent from the computer associated with thedisplay on which the change has been made to the other computer(s)viewing that object. This allows very fast performance once the objecthas been initially stored, as only the changed data is communicated.Further, this allows the change to be illustrated in real time, e.g.,the remote display will show the zoom or pinch rather than abruptlychange the size or will show the annotation as it is being drawn, ratherthan once a line or character is completed.

As noted above before this discussion of how mobile devices interactwith

Canvas, whether collocated or at one or more different locations, whenthe Canvas was manipulated, the video conferencing stream may beaffected, e.g., may disappear altogether from the display region.However, users may want the video conferencing stream, e.g., a livevideo feed of a remote user, to remain on the display region, e.g., at asame position on the display region, at all times. Therefore, accordingto embodiments set forth below, a live video feed may be treateddifferently than other windows on the canvas.

For example, as illustrated in FIG. 3, when the video conferencing panel(VCP) is streamed for a remote device 300 to the Common Display 110 at asingle location, rather than placing the VCP on the Canvas, the VCP maybe in a particular position on the Common Display 110. The remote device300 may provide the live video stream using a peer-to-peer connection.This peer-to-peer connection may be established through a relay server,e.g., in the cloud, which does not store anything, but just relays thatlive video stream. The VCP may originally appear as a thumbnail in thetray region 114. The VCP may be moved, resized, and so forth, withoutaffecting the remainder of the Canvas and the Canvas may be manipulatedas discussed above with affecting the VCP. The VCP may always be broughtto the front of the display such that a portion of the Virtual Canvasunder the VCP will not be visible. Having established the connection tothe Virtual Canvas on both Common Displays, e.g., having started asession, no additional information is needed to start the videoconferencing, i.e., other than selecting the software to be used for thevideo conference, no separate video conferencing session is needed. Inan implementation, the Virtual Canvas may be stored in the cloud, whilethe video conferencing would still be peer-to-peer, as it is separatefrom the Virtual Canvas.

When using multiple locations, the VCP may be streamed from a firstlocation to the other multisite locations, as shown in FIG. 4 and viceversa. This allows users in the various other locations to view thevideo in real time, i.e., at the same time as the source location (realtime video viewing is enabled with minimal inherent delay due to thepeer to peer connection). Again, the VCP is to be separate from theCanvas. Thus, each location may have a VCP at a different positionand/or size on the corresponding Displays and viewing other location(s)video streams VCW1 and VCW2 (and may display its own video conferencestream in a VCW), but still see identical views of the Canvas havingother windows thereon. Here, the live video may stream directly betweenlocations (direct peer-to-peer connection) or may stream through a relayserver forming a peer-to-peer connection. When there are more than twolocations streaming video, each VCW for the other locations may havetheir own positions that do not change with movement or othermanipulation of the Canvas.

When forming a peer-to-peer connection through a relay server, the relayserver may be on the same internal network as the Multisite computers(private network connection) or may be in the cloud. If the Relay Serveris placed on the private network, then the speed of the connection maybe faster compared to a server placed in the cloud. However, even aRelay Server placed in the cloud is typically much faster than thetypical video conference method, as described above.

In particular, when using Multisite, all objects on the canvas are sentbetween computers. The Common Display may display the VCP in one sectionof the Display and the Virtual Canvas in another section. VCP showingthe live video conferencing feeds, e.g., windows in the VCP, may be on apanel at a particular location on the Common Display, e.g., on a sideportion of the Common Display, as shown in FIG. 5. All other video feedsmay be on the Virtual Canvas. This way when the Virtual Canvas is movedor resized on the Common Display, the VCP remains stationary.

As shown in FIG. 5, the VCP may include multiple video conferencingwindows (VCWs), e.g., VCW1 to VCW2. When one of the VCWs is to beselected, e.g., either an audio signal is associated with the window ora user of the Common Display selects a VCS, here VCS2, that VCS may bealtered, e.g., may be closer to a camera and/or enlarged to enhance theconferencing experience. For example, when the selected VCS changes tobe near the camera, the user of the Common Display will more closelyappear to be interacting with the user in the VCS.

In contrast, when using AirConnect™ on a mobile device, a user of themobile device views a rendering of a canvas, rather than the objectsbeing sent thereto. Here, as shown in FIGS. 7 and 8, one window containsthe canvas and there is a separate panel VCP for video conferencing. TheVCP may have a primary large window and a strip, e.g., on the right sidethat contains multiple smaller windows. Each of the smaller windowscontains a video stream from a remote user. Clicking on any one of thesmaller windows, e.g., VCW2 in FIG. 6, places the corresponding videostream in the large window. Alternatively, several windows may be largerand several windows smaller. Another alternative is for the AirConnect™app to detect which window contains audio and for that window to beautomatically made larger. Since a camera 450 on the mobile device istypically centrally located and the mobile device screen is much smallerthan the Common Display, moving the VCW may not be needed.

Mobile Device-to-Device Video Conferencing: Once a single user downloadsan AirConnect™ app on to their mobile computer device, they may requestto schedule a video conference with a particular user. Typically eachuser will have a user account. A cloud server can be used to schedule avideo conference between two users with User Accounts. Once the videoconference is initiated then the video conference can be started with apeer-to-peer connection. The virtual canvas data can be stored on adevice (Host Computer) that is running the ThinkHub® app. The HostComputer can be one of the mobile computers in the video conference or aseparate server. If the Host Computer is a separate server, it may be acloud server. The Host Computer may be the Display Computer. But in thiscase a physical display is not necessary, since there may be no need fora user to be located near this display, in a position where they can seeor touch this display. In this case, the Host Computer a Virtual DisplayComputer.

Alternatively, without user accounts, a first user can initiate a videoconferencing call with the AirConnect™ app. Once the video conference isestablished a key can then be generated a first user can then contact asecond user and send them the key. The second user can then joint theVideo Conference by entering the key into the AirConnect™ app. As in thepreceding paragraph, the Host Computer can be one of the mobilecomputers in the video conference or a separate server.

Mobile Device to Display Computer Video Conferencing. In this case theremay be one or more users in a conference room working on a touch displaythat is being driven by a Display computer. A user located remotely isworking on a Mobile Device Computer. This remote user has AirConnect™app downloaded and running on their Mobile Device Computer. The remoteuser joins the ThinkHub® session running on the Display Computer throughthe AirConnect™ app. This allows the participants in the conference roomwith the Display Computer to start a video conference between them andthe remote user.

Multisite Video Conferencing: This for video conferencing betweenmultiple rooms, where each room has a Display Computer and explainedabove.

Remote Users joining a Multisite Video Conference. Suppose there are two

Display Computers in a Multisite Video Conferencing session (DisplayComputer 1 and Display Computer 2). If a remote user joins DisplayComputer 1 in a video conference, then the video feed from the camerafrom the remote user will be sent to Display Computer 1. Then theThinkHub® multisite application will send that same video feed toDisplay Computer 2. Display Computer 1 and Display Computer 2 will sharevideo and data as in the preceding paragraph. However, in this case bothDisplay Computer 1 and Display Computer 2 will send the videoinformation from their camera to the remote user. Alternatively, eachvideo stream may be sent to every display individually, rather thanbeing sent through the Multisite connection.

For either configuration, for further data reduction any of the aboveembodiments, down sampling (Dynamic Resolution Scaling (DRS)) may beused as disclosed in U.S. patent application Ser. No. 15/056,787 and inInternational Application No. PCT/US2017/064525, both of which areincorporated herein in their entirety for all purposes. When streaminglive video windows in the manner disclosed above the bandwidth requiredcan be much larger than with conventional web-based or videoconferencing. For example, suppose at location 1 there are six videosources connected to COMP1. Each of these may be placed on the canvasand each may have a resolution of 1080p hd. Suppose that at location 1,DISPLAY1 has a resolution capable of displaying all six video sourcessimultaneously and the view of the canvas is such that all six videowindows are displayed on DISPLAY1 at the same time in full 1080p HD.(That would mean that DISPLAY1 has resolution of at least 12k).Normally, in web-based conferencing the source computer sends the entirevideo stream at a predetermined resolution because it has no informationof how the content is being viewed on the receiving side. However, withThinkHub® multisite software, if at location 2, COMP2 has only a 4kdisplay, then, at most 4k resolution would need to be transmitted.Suppose for example at location 2, the canvas is being zoomed in to aparticular section of the canvas such that only three of the six videosare in view at a given time and these three videos are being displayedat 720p resolution. Then COMP2 can send data to COMP1 requesting thatthe three videos in view are sent at full frame rate but at a lowerresolution (720p). The three videos out of view can be sent at lowerresolution and/or very low frame rate, for example 1 frame per 2seconds. In this manner the total bandwidth can be reduced to lower thanthe size of the display at location 2. In this manner, the bandwidth ofthe data transmitted can be reduced to the lower of

-   -   (a) The resolution of the receiving display times the viewing        frame rate;    -   (b) The combined resolution of all of the streaming video        signals times the viewing frame rate; and    -   (c) The combined resolution of all of the streaming video        signals as displayed on the receiving display times the viewing        frame rate.

The algorithm may be set to always lower the bandwidth to the lower of(b) and (c) above, which will mean that it will always be as low orlower than the lower of (a) and (b) above. Note that in the aboveanalysis bandwidth for the off-screen video signals was not taken intoaccount, since they would be sent at such low frame rate that theirbandwidth would be negligible.

In the method above, all the video sources send all info at fullresolution to COMP1. Based on information received from the othercomputers in the Multisite session on how the live video windows arebeing viewed thereon, COMP1 down samples the transmission to the othersites. In addition to this, COMP1 can also use this information to sendinfo to the live video sources to downsample the video streams prior tosending the info to COMP1. In this situation a laptop would only sendits full resolution signal if at least one computer in the multisitesession was viewing that live signal at the laptop's full resolution orlarger.

In accordance with another embodiment, a display computer 120 in FIG. 1may run a restricted Virtual Canvas, e.g., HubVC™ video conferencing andcollaboration software by TIV®. While the Virtual Canvas described abovein connection with the first embodiment is unrestricted, i.e., verylarge, e.g., twenty or up to infinite times the viewable portion of thedisplay, provides great flexibility and capabilities, some users thatjust want to do a few simple tasks or work on only a small number ofdocuments at the same time, the use of the unrestricted Virtual Canvasmay unnecessarily complicate use thereof. Thus, a restricted VirtualCanvas may provide simpler and limited set of features, i.e., somefeatures of the unlimited Virtual Canvas are eliminated or restricted inorder to make it easier to use and cheaper. For example, windows in theRestricted mode cannot be panned, zoomed or pinched as described abovefor the unrestricted Virtual Canvas. The restricted Virtual Canvas maystill serve as a shared canvas when another computer is running canvassoftware, e.g., ThinkHub® canvas software or HubVC™ canvas software.

In one example of the restricted Virtual Canvas, only a few views of therestricted Virtual Canvas are allowed. For example, there may be twoallowed views, e.g., a single view or minimum view, i.e., where a fixednumber e.g., four, of windows are displayed, with the size of eachwindow being a minimum size to be displayed. The fixed number of windowsthat are in view makes interaction therewith more manageable, limits theamount of processing power needed, and insures a minimum size of awindow. These windows may each display one of a mobile device stream, awhite board, a video conference stream, and a snapshot of a stream, andwindows having at least two of different types may be on the restrictedVirtual Canvas. While numerous windows, may be on the restricted VirtualCanvas, only a fixed maximum number, e.g., four, windows may be viewedat a time. When more streams or windows than the fixed number are sentto or opened on the restricted VirtualCanvas, they can be placed on therestricted VirtualCanvas, out of view and therefore not displayed, theymay still be accessed by scrolling or a small portion of theout-of-display windows can be displayed to indicate to a user that thereis more content on the restricted Virtual Canvas out of view.

In this manner, if the use of the restricted Virtual Canvas is limitedto only a single view and minimum view, e.g., a quad view, there aresome advantages in terms of simplicity and ease of use. For example,with an unrestricted Virtual Canvas, if many pieces of content areloaded thereon and the unrestricted Virtual Canvas is zoomed out to viewthe entire unrestricted Virtual Canvas or a large portion thereof, itcan be difficult for a user to view the individual windows.

The HubVC™ Canvas ensures that the main windows are never smaller than1/n the size of the full screen, wherein n is the maximum number ofwindows to be viewed, so that all of the windows can be easily viewed.

Comparisons between an unrestricted Virtual Canvas and a restrictedVirtual Canvas are illustrated in FIGS. 9A to 9C. As may be seen in FIG.9A, when content is on the unrestricted Virtual Canvas, windowsdisplaying the content may have various sizes and may not be visible inthe view, indicated by the dashed line, of what is actually shown on thedisplay. As may be seen in FIGS. 9B and 9C, when content is on therestricted Virtual Canvas, all windows have the same size. When therestricted Virtual Canvas is in the single view mode, as shown in FIG.9B, only a single window (or a single window and edges of adjacentwindows) is viewable in the display. When the restricted Virtual Canvasis in the fixed number view mode, e.g., quad mode, as shown in FIG. 9C,only a fixed number of windows (or a single window and edges of adjacentwindows) are viewable in the display. This is true even if only twostreaming content are being sent, with the other windows remainingblank. In both embodiments, windows that are outside of what is actuallybeing displayed may not be updated in real time.

More detailed examples of operation of the restricted Virtual Canvas areillustrated in FIGS. 10A to 10U. Details of functionality of theunrestricted Virtual Canvas are disclosed in U.S. Pat. No. 9,596,319,incorporated herein by reference for all purposes. Further, any of thefunctions described below in connection with the restricted VirtualCanvas may be used with the first embodiment of the unrestricted VirtualCanvas. Additionally, while different configurations of the windows,control panels, display, etc., may be shown in different drawingfigures, the same functionality may be provided in any of theseconfigurations.

As may be seen in FIG. 10A, before any live streams are provided, aninitial screen may be displayed that includes a white board interface1010, video conference interfaces 1020 a to 1020 d, e.g., a separateinterface for each supported video conferencing software, a share yourscreen interface 1030, e.g., a QR code to scan and a web address. A listof future or current meetings may be displayed, e.g., in a region 1050above the interfaces.

For example, as shown in FIG. 10B, when the white board interface 1010is selected, a whiteboard may occupy most of the screen. A control panel1100 may be provided, e.g., at a lower portion of the screen. Thecontrol panel 1100 may include a touch mode interface 1110, anannotation mode interface 1120, a trash mode interface 1130, awhiteboard interface 1140, and a video conference interface 1150. If theannotation mode interface 1120 is selected when no windows are on therestricted Virtual Canvas, a whiteboard window may be automaticallyopened Each of these modes will be described below.

When a first piece of content is sent to the restricted Virtual Canvas,the restricted Virtual Canvas may be zoomed in to display only a singlewindow containing this first piece of content (Single View), e.g., thewhiteboard may be replaced by the content window, as shown in FIG. 10C.Also, as shown in FIG. 10C, a control panel 1100 a may be provided onone side of the screen and may include the annotation mode interface1120, the trash mode interface 1130, the whiteboard interface 1140, thevideo conference interface 1150, and expand to full screen interface1160. Alternatively, the restricted canvas may be a minimum mode,whiteboard may be maintained in one of the windows and the contentwindow in another of the fixed number of windows.

At another location on the display, e.g., a top of the display,information about the display, e.g., a name of the display 1210, an IPaddress 1220, and 1230. Further, other screens available on therestricted Virtual Canvas may be indicated in scroll window 1240. Aleave session button 1250 will terminate the session for only the screenbeing displayed. A toggle icon 1260 allows the view to be toggledbetween a quad mode by selecting the four window icon and a single modeby selecting the single window icon, as shown.

An icon 1270 may provide an accessibility mode. When selected, all ofthe information that was at the top of the display is moved to thebottom of the display and the control panel 1100 a may move from amiddle of the left hand side of the display to a lower left right handside of the display, as shown in FIG. 10D. Selecting the icon 1270 againwill return to the default state shown in FIG. 10C.

When a second piece of content is added to the restricted VirtualCanvas, the view switches to a minimum view having the fixed number ofwindows, as shown in FIG. 10E. The remaining windows may be empty withsome type of framing, may not be evident, may include whiteboards, maydisplay the initial screen, and so forth. In an implementation, therestricted Virtual Canvas may be forced to stay in single view, with themost recently added content automatically replacing the previouslydisplayed content.

In annotate mode, any of the whiteboards or stream may be annotated on,as shown in FIG. 10E. As shown therein, when the annotation modeinterface 1120 is selected, a detailed menu 1125 related to annotationmay appear.

When the video conference interface 1150 is selected, a videoconferencing panel 1300 may occupy half of the screen and may include awindow displaying the different video conference interfaces 1020 a to1020 d to be selected, as shown in FIG. 10F. As further shown therein,the windows may be slid to the right, so that the size of the firstupper left window having the stream and the lower left window having theright board are visible. The control panel 1100 a may now be between thevideo conferencing panel 1300 and the windows. Selecting an end videoconference interface 1320 will end all video conferences in the videoconferencing panel 1300. Individual video conference windows VCW1 toVCW3 may include individual interfaces, e.g., an “x”, to selectively endindividual video conferences. Unlike the windows on the restrictedvirtual Canvas, the windows in the video conferencing panel may befreely moved and resized.

In expand to full screen mode, as shown in FIG. 10G, the individualwindows may be expanded to go to edges of the display screen, i.e.,without information in an upper portion being visible, as compared withFIG. 10I. Further, if the minimum mode of FIG. 10I was expanded to fullscreen, the gray borders there between would be eliminated as well asthe upper portion. Further, this particular implementation may use acontrol panel 1100 b that includes the annotation mode interface 1120,the trash mode interface 1130, a touch mode interface 1110, and thetoggle icon 1260 to select the different view modes. The videoconference may be controlled from the video conferencing panel 1300 asshown in FIG. 10F. Once the video conference is selected, the videoconferencing panel 1300 may always be visible.

The video conferencing panel 1300 may be expanded up to a certainportion of the screen, e.g., half of the screen, as shown in FIG. 10H.This may be done by dragging the video conferencing panel 1300 or usingan expand button. When in single view mode of FIG. 10H, the aspect ratioof the window may be maintained, such that a portion below the window isempty. Thumbnails of other windows may be shown therein. These may notbe updated in real time, reducing the burden.

In touch mode, the windows may be rearranged, as shown in FIGS. 10I to10K, or a window off to the side may replace one of the windows beingdisplayed, as shown in FIGS. 10L to 10M. For simplification, thescrolling between windows may be along a single dimension, e.g.,laterally, e.g., left and right as shown here, or vertically, e.g., upand down. In other words, the minimum mode may have the windows arrangedin a matrix of x rows and y columns, wherein each of x and y are 2 orgreater, one of x and y being fixed, e.g., to two, and another of x andy is larger, e.g., unlimited.

In the delete mode, as shown in FIG. 10N, each window being displayedmay be provided with a delete button so that one or more of thesewindows may be deleted. Also, as may be seen in FIG. 10N, edges ofscreens adjacent to those being viewed may be seen off to respectivesides.

In an implementation shown in FIGS. 10O and 10P, instead of a simpledelete mode, a control panel 1100 c may include an edit mode interface1180 may be provided, which may include dragging as well as deleting andtaking a snapshot when that mode is selected as shown in FIG. 10P. Asalso shown in FIGS. 10O to 10Q, each window may include a toggleinterface 1280 to switch between the minimum mode display of FIGS. 10Oand 10P to the single display mode of FIG. 10Q and back again.

Referring again to FIGS. 10E and 10F, a useful function to have on boththe unrestricted and restricted Virtual Canvases is a snapshot function.This is particularly useful for a live stream (e.g. a lap top feed, aweb browser, etc.). Selecting a snapshot function using the snapshoticon 1290 in each window in a particular first window on the restrictedVirtual Canvas would produce a snapshot in a second window of theinstantaneous view of the content of the first window. The snapshot inthe second window would then be a static image, so that even if thefirst window changes, the second window would remain with the samestatic image. The second window would then also have all of the optionsavailable in the other windows, including the ability to annotate on topof the snapshot. When a snapshot is taken in the unrestricted VirtualCanvas, the snapshot may correspond to the size of the object and may beindependently moved and resized, as any other object in the unrestrictedVirtual Canvas.

As illustrated in FIG. 10R, when one of the video conference interfacesis selected, a video conferencing panel 1300 may occupy a fixed portionof the display, e.g., half of the screen. Here, the control panel 1100 amay be between the video conferencing panel 1300 and remaining windowse.g., two windows that are maintained in the minimum mode size. In animplementation, as shown in FIG. 10M, a position of the control panel1100 a may remain unchanged. The video conferencing panel 1300 mayinclude the video conference interfaces 1020 a to 1020 d to allow a userto select which VC provider to be used, e.g., a native AirConnect™ orthird party video conferencing software, e.g., Zoom™, Skype™, WebEx®,and so on. When the video conferencing panel 1300 occupies half of thescreen, only a subset of the windows, e.g., two, of the windows inminimum mode may be viewed, so the size thereof is not reduced. When insingle mode, the window will be reduced to occupy half of the screen.

Alternatively, the video conferencing panel 1300 may occupy a smallerfixed portion of the display, as shown in FIG. 10S. When the videoconferencing panel VCP occupies less than a predetermined amount of thescreen, e.g., a quarter of the screen, the windows in the minimum modemay be slightly reduced or a right most edge of the right most windowsmay be cropped. When the video conferencing panel VCP occupies more thanthe predetermined amount of the screen, only two of the windows may beshown.

As a further alternative, as shown in FIGS. 10T, the video conferencingpanel 1300 may be resizeable and movable and remains on top of thewindows. Thus, in all of these, the video conferencing panel 1300 is notpart of the restricted Virtual Canvas.

As a further alternative, as shown in FIG. 10U, unlike videoconferencing on an unrestricted Virtual Canvas, as described above,since the number of windows in view is fixed, the video conference mayoccupy a predetermined window of the windows in minimum mode, e.g.,VCW1and VCW2. VCW1 may show the stream being output from common displaycomputer and VCW2 may be a stream output from a remote computer. Unlikethe video display panel discussed above, only a single video conferencestream may be displayed in each window. Since windows on the restrictedVirtual Canvas cannot be zoomed in and out, like in the Virtual Canvas,one of the windows may be readily used for the video conference.Further, while the video conference windows be moved, annotated orotherwise treated like any other window, the windows shown in FIG. 10Umay include, in addition to the toggle interface 1280, a pin interface1295 that prevents the position of the window from moving even whenother windows are scrolled. When the pin interface 1295 is selected,that window, e.g., VCW2, may stay in view.

Embodiments are described, and illustrated in the drawings, in terms offunctional blocks, units and/or modules. Those skilled in the art willappreciate that these blocks, units and/or modules are physicallyimplemented by electronic (or optical) circuits such as logic circuits,discrete components, microprocessors, hard-wired circuits, memoryelements, wiring connections, and the like, which may be formed usingsemiconductor-based fabrication techniques or other manufacturingtechnologies. In the case of the blocks, units and/or modules beingimplemented by microprocessors or similar, they may be programmed usingsoftware (e.g., microcode) to perform various functions discussed hereinand may optionally be driven by firmware and/or software. Alternatively,each block, unit and/or module may be implemented by dedicated hardware,or as a combination of dedicated hardware to perform some functions anda processor (e.g., one or more programmed microprocessors and associatedcircuitry) to perform other functions. Also, each block, unit and/ormodule of the embodiments may be physically separated into two or moreinteracting and discrete blocks, units and/or modules without departingfrom the scope of the inventive concepts. Further, the blocks, unitsand/or modules of the embodiments may be physically combined into morecomplex blocks, units and/or modules without departing from the scope ofthis disclosure.

The methods and processes described herein may be performed by code orinstructions to be executed by a computer, processor, manager, orcontroller. Because the algorithms that form the basis of the methods(or operations of the computer, processor, or controller) are describedin detail, the code or instructions for implementing the operations ofthe method embodiments may transform the computer, processor, orcontroller into a special-purpose processor for performing the methodsdescribed herein.

Also, another embodiment may include a computer-readable medium, e.g., anon-transitory computer-readable medium, for storing the code orinstructions described above. The computer-readable medium may be avolatile or non-volatile memory or other storage device, which may beremovably or fixedly coupled to the computer, processor, or controllerwhich is to execute the code or instructions for performing the methodembodiments described herein.

Example embodiments have been disclosed herein, and although specificterms are employed, they are used and are to be interpreted in a genericand descriptive sense only and not for purpose of limitation. In someinstances, as would be apparent to one of ordinary skill in the art asof the filing of the present application, features, characteristics,and/or elements described in connection with a particular embodiment maybe used singly or in combination with features, characteristics, and/orelements described in connection with other embodiments unless otherwisespecifically indicated. Accordingly, it will be understood by those ofskill in the art that various changes in form and details may be madewithout departing from the spirit and scope of the present invention asset forth in the following claims.

What is claimed is:
 1. A method of collaborating between a firstcomputer associated with a first display at a first location and asecond computer associated with a second display at a second location,the method comprising: establishing a connection between the first andsecond computers; opening a virtual canvas on the first computer, thevirtual canvas to be displayed on the first and second displayssimultaneously; and sending an object between the first and secondcomputers by sending data associated with the object on the virtualcanvas stored on the first computer from the first computer to thesecond computer to be stored locally, thereby creating a shared canvas,wherein objects are at a single location on the shared canvas, sending afirst live video stream from the first display to the second display tobe viewed in a first video conference window separate from the virtualcanvas on the second display, and sending a second live video streamfrom the second display to the first display to be viewed in a secondvideo conference window separate from the virtual canvas on the firstdisplay.
 2. The method as claimed in claim 1, further comprisingconnecting a mobile display device to one of the first and secondcomputers, sending a rendering of the shared canvas, and directlysending the first and second live video streams in a video conferencewindow on the mobile display device separate from the rendering.
 3. Themethod as claimed in claim 1, wherein each of the first and second videoconference windows is in a video conferencing panel.
 4. The method asclaimed in claim 3, further comprising: selecting a video conferencewindow; and altering the selected video conference window.
 5. The methodas claimed in claim 4, wherein altering the selected video conferencewindow includes moving the selected video conference window to beadjacent a camera on a display viewing the window.
 6. The method asclaimed in claim 4, wherein altering the selected video conferencewindow includes enlarging the window.
 7. The method as claimed in claim4, selecting the video conference window includes detecting an audiosignal associated with the video conference window.
 8. The method asclaimed in claim 3, wherein the video conferencing panel is stationaryand the at least two video conference windows are manipulatable.
 9. Themethod as claimed in claim 1, wherein first and second video conferencewindows are stationary, and the shared canvas and objects thereon aremanipulatable.
 10. The method as claimed in claim 9, wherein each of thefirst and second video conference windows are adjacent a cameraassociated with a display on which respective video conference windowsare viewed.
 11. The method as claimed in claim 9, wherein the videoconference window is in a periphery of a display on which respectivevideo conference windows are viewed.
 12. The method as claimed in claim1, wherein the connection for the virtual canvas provides informationneeded to send the first and second live video streams.
 13. The methodas claimed in claim 1, wherein the connection for the virtual canvasprovides information to send the first and second live video streams.14. The method as claimed in claim 1, wherein the virtual canvas isstored in the cloud and the first and second live video streams aretransmitted in real-time.
 15. The method as claimed in claim 1, whereinthe virtual canvas is a restricted virtual canvas.
 16. The method asclaimed in claim 1, wherein: the first live video stream is displayed inthe second video conference window separate from the virtual canvas onthe first display, and the second live video stream is displayed in thefirst video conference window separate from the virtual canvas on thesecond display.
 17. A method of collaborating between first and secondmobile devices having sharing application downloaded thereon and a hostcomputer, the second mobile device being remote from the first mobiledevice and the host computer, the method comprising: establishing apeer-to-peer connection between the first and second mobile devices andthe host computer; opening a virtual canvas on the host computer havinga host display, the virtual canvas to be shared by the first and secondmobile devices and the host display; sending a first object from thefirst mobile device to the virtual canvas by sending data associatedwith the first object to the host computer and sending the first objectto the second mobile device by sending data associated with the firstobject from the host computer to the second mobile device; sending afirst live video stream from the first mobile device to the host deviceto be viewed in a first video conference window separate from thevirtual canvas on the host display, and sending a second live videostream from the second mobile device to the first mobile device to bethe host device to be viewed in a second video conference windowseparate from the virtual canvas on the host display.
 18. The method asclaimed in claim 17, wherein the virtual canvas is stored in the cloud.19. The method as claimed in claim 17, wherein collaboration is toinclude a second host computer having a second host display located nearthe second mobile device, the method further comprising: establishing apeer-to-peer connection between the host computer and the second hostcomputer, wherein the virtual canvas to be displayed on the first andsecond displays simultaneously; and displaying the first and second livevideo streams on the second host display in first and second windowsseparate from the virtual canvas.
 20. The method as claimed in claim 17,wherein sending the first and second live video streams includes usinguser accounts associated with the first and second mobile devices. 21.The method as claimed in claim 17, wherein establishing the peer-to-peerconnection includes: generating a key by the host device, reading thekey by the first mobile device; sending the key to the second mobiledevice, and entering the key into the second mobile device.
 22. A methodof collaborating between a first computer associated with a firstdisplay at a first location and a second computer associated with asecond display at a second location, the method comprising: establishinga connection between the first and second computers; opening a virtualcanvas on the first computer, the virtual canvas to be displayed on thefirst and second displays simultaneously; and sending an object betweenthe first and second computers by sending data associated with theobject on the virtual canvas stored on the first computer from the firstcomputer to the second computer to be stored locally, thereby creating ashared canvas, wherein objects are at a single location on the sharedcanvas, sending a first live video stream from the first display to thesecond display to be viewed in a first video conference window on thesecond display, and sending a second live video stream from the seconddisplay to the first display to be viewed in a second video conferencewindow on the first display, wherein at least one of the first andsecond video conference windows is not on the virtual canvas.